Multifractal analysis aims to characterize signals, functions, images or fields, via the fluctuations of their local regularity along time or space, hence capturing crucial features of their temporal/spatial dynamics. Multifractal analysis is becoming a standard tool in signal and image processing, and is nowadays widely used in numerous applications of different natures. Its common formulation relies on the measure of local regularity via the H\"older exponent, by nature restricted to positive values, and thus to locally bounded functions or signals. It is here proposed to base the quantification of local regularity on $p$-exponents, a novel local regularity measure potentially taking negative values. First, the theoretical properties of $p$-exponents are studied in detail. Second, wavelet-based multiscale quantities, the $p$-leaders, are constructed and shown to permit accurate practical estimation of $p$-exponents. Exploiting the potential dependence with $p$, it is also shown how the collection of $p$-exponents enriches the classification of locally singular behaviors in functions, signals or images. The present contribution is complemented by a companion article developing the $p$-leader based multifractal formalism associated to $p$-exponents.